Evaluation Highlights 

Client Feedback from October 2006 - September 2009 

compiled December 2009 

EVALUATION PI AN AND METI TOPS 

The NECTAC Evaluation Unit collects and 
analyzes data to describe the quantity and nature of 
the technical assistance services provided by 
NECTAC staff. The Unit determines the quality 
and relevance of NECTAC services and products 
and whether clients use NECTAC Technical 



TYPE OF TA 


TA Provider 
Time (hours) 


% of Total 
Time 


Individualized TA 


10,048.25 


0.74 


Cross-State TA 


3577.00 


0.26 


Major TA 


9546.00 


0.70 


Less Extensive TA 


4079.25 


0.30 




Assistance (TA) to improve state and local systems 
and services. An external evaluation was 
conducted with all primary clients (Part C and 
Section 619 Coordinators from all states and 
jurisdictions) regardless of their level of 
participation in NECTAC TA. The highlights 
included in this report include data from October 1 , 
2006 to September 30, 2009. 

MAJORF1NDINGS - IXXXJMENTAIION 

NECTAC staff provided more than 13,000 hours 
of direct TA to Part C and Section 619 programs in 
all 60 states and jurisdictions. Seventy-four 
percent (74%) of that time was spent on 
individualized TA where the TA is focused on a 
single state with a specific issue or TA request. 
The other 26% was spent on cross-state TA where 
the TA is designed for many or all states, such as 
OSEP’s National Early Childhood Conference. 

Seventy percent (70%) of the total time expended 
in NECTAC TA efforts was spent providing 
“Major TA” services which were generally more 
than 8 hours and anticipated long term impacts for 
improving systems and/or practices. The other 
30% was spent providing “Less Extensive” 
services in response to client requests. 



From October 1, 2006 to September 30, 2009, the 
NECTAC staff provided a total of 2,165 technical 
assistance services on a range of topics. The areas 
receiving the most effort to date are summarized in 
the following table. 



TOPIC 


TA Provider 
Time (hours) 


1. Developing and implementing 
systems for measuring child and 
family outcomes. 


3068.00 


2. Expanding opportunities for 
young children with disabilities to 
receive services in 
Inclusion/LRE environments 
with their peers. 


2130.25 


3. Developing SPPs/APRs and 
developing data systems for 

collecting and reporting data. 


1971.75 


4. Implementing evidence based 
El services, including IFSP 
process, service coordination and 
natural environments. 


1789.75 


5. Building effective general 
supervision and monitoring 
systems to ensure compliance 
with IDEA and quality services. 


1584.75 


6. Improving the transition process 
as young children move from 
Early Intervention to Preschool. 


1163.75 
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TA&D 



NETWORK 



NECTAC provided TA to a total of 14,899 state 
clients including state Part C and Section 619 
administrators; staff from institutes of higher 
education, state TA providers and CSPD 
coordinators responsible for personnel 
development; regional and local administrators for 
early intervention and preschool programs; direct 
service providers including teachers, early 
interventionists, therapists; and family members 
and family representatives. The pie chart below 
depicts the proportion of state participant types, by 
system level, included in all NECTAC TA 
Services. 



Proportion of Participant Types Included 
in All NECTAC TA Services 
By System Level 



■ State 
Administrators 

□ Service 
Providers 

■ Regional/Local 
Admini stators 

□ Personnel 
Development 

■ Family 
Representatives 




NECTAC also provided TA to a variety of other 
participants including federal agency staff from 
various programs, professional organizations, 
national and regional TA providers, private 
consultants, and the media. 

NECTAC staff assisted states in developing and 
implementing 22 long-term systems change 
initiatives on topics such as developing systems for 
measuring child and family outcomes, building 
quality inclusive services/systems, ensuring 
smooth EC transitions and building/improving 
sustainable finance systems. Systems change 
initiatives are based on an ongoing relationship 
with a state and the work is driven by a 
stakeholder-developed strategic plan for improving 
the state system for implementing IDEA. 



CLIENT SATISFACTION 



The external evaluation was sent to all clients 
regardless of the extent to which they participated 
in NECTAC services. For every NECTAC TA 
service evaluated, participants are asked to rate the 
quality of the service, the relevance of the 
information and activities, and the provider 
expertise on a scale of 1 to 5 (5 being the highest 
rating). The evaluation also asked clients about 
their overall perceptions of the quality and 
relevance of NECTAC services and products. 
Below is a summary of the mean ratings related to 
these three GPRA indicators. 



TYPE OF TA 


High 

Quality 


Relevance 


Provider 

Expertise 


Less Extensive 


4.7 


4.4 


4.8 


Major 

Individualized 


4.5 


4.6 


4.6 


Cross-State 


4.2 


4.4 


4.5 


Overall 


4.4 


4.4 


4.6 


Products* 


4.3 


4.2 


n/a 


Overall* 


4.4 


4.2 


n/a 



*External Evaluation results 



Quality 

The external evaluation also revealed that a strong 
majority of clients (96%) agreed that the services 
and products they received from NECTAC were of 
high quality, typically citing NECTAC ’s prompt 
response to inquiries and accuracy when sharing 
technical information. With regard to specific 
products such as NECTAC ’s Contact Lists, Part C 
Updates, Section 619 Profile, the NECTAC 
website, eNotes and client listservs, over 97% of 
clients said they agree or strongly agree that what 
they received is of high quality. 

Relevance 

Additionally, a majority of clients (92%) agreed 
that the services and products they received from 
NECTAC were relevant to their policies or 
practices, typically citing NECTAC ’s ability to 
share information from other states and awareness 
of OSEP requirements. Ninety-seven percent 
(97%) agreed that NECTAC ’s electronic resources, 
website, 

eNotes and client listservs are relevant to their 
policies and practices. 
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PROJECT EFFECTIVENESS 



Immediate Outcomes 

The Evaluation Unit measured immediate and 
longer term outcomes to determine the 
effectiveness of TA. The accomplishment of 
immediate outcomes for each event is measured to 
determine the extent to which the TA was able to 
increase the participants’ capacity to implement 
quality EI/ECSE programs and/or practices. 
Immediate outcomes included increasing 
participant knowledge, skills, access to resources, 
changing attitudes and developing action plans. 
The overall mean ratings for accomplishment of 
intended outcomes by type of TA are listed below: 



TYPE OF TA 


Overall Mean for 
Intended Outcomes 


Less Extensive TA 


4.5 


Major Individualized TA 


4.5 


Cross-State TA 


4.1 


Overall 


4.4 



Some examples of participant feedback related to 
the accomplishment of immediate outcomes: 

“We will use the information to develop criteria for 
exemplary programs statewide. Those programs 
will then become visitation site models for others to 
replicate. The information will also be shared at 
our statewide leadership symposium. ” 

“[We gained] techniques to use and share with 
staff. Some staff members fear data collection, but 
some of the ideas shared by presenters and 
colleagues demonstrated data collection can be 
painless !” 

“ The training has given me a deeper focus on the 
importance of teamwork to more accurately 
determine progress and positive outcomes for 
young children with disabilities. ” 

Long-Term Impacts 

Long term impacts were measured by NECTAC 
through follow up surveys and interviews with 
states. In addition, an external evaluation was 
conducted with all primary clients. Both internal 
and external evaluations revealed impacts on state 
and local systems and on practices. 

State System Level Changes 

Ninety-one percent (91%) of external evaluation 
respondents agreed or strongly agreed that their 
state level infrastructure had been improved 
because of the services and/or products received 



from NECTAC. States reported changes in the 
following areas: 

• Guidance (84%); 

• Policies and procedures (73%); 

• General supervision/monitoring (40%); and 

• Inter-agency relationships (3 1 %) 

Data collected by NECTAC as follow-up to Major 
Individualized TA and Long-Term Systems 
Change Initiatives showed similar results. Ninety- 
five percent (95%) of Major Individualized TA and 
97% of Systems Change Initiatives resulted in 
changes in state systems, as reported by key 
contact(s) in the state. Some examples of client 
feedback related to state level changes as a result 
of NECTAC TA: 

“ NECTAC was able to help us assess our general 
supervision system so that we have the ability to 
timely identify non-compliance policies that needed 
clarification for personnel, or develop targeted 
trainings, and ultimately implement changes to 
practices to achieve improvement. ” 

“[We are] bringing key folks with decision making 
capacity together to problem solve and set up 
strategies to support early childhood transition. 
Commitments and implementation have begun in 
the area of data sharing and finalizing a state level 
memorandum of understanding. ” 

“Information was gained at the conference that 
definitely helped us in completing our APR in 
February. We’ve also used information from the 
conference to aid us in redesigning our TA system 
to local early intervention programs. ” 

Local System Level Changes 

Ninety-one percent (91%) of external evaluation 
respondents agreed or strongly agreed that their 
local level infrastructure had been improved 
because of the services and/or products received 
from NECTAC. States reported changes in these 
areas: 

• Guidance (72%); 

• Policies and procedures (50%); 

• Inter-agency relationships (33%); and 

• General supervision/monitoring (31%) 

Data collected by NECTAC as follow-up to Major 
Individualized TA and Long-Term Systems 
Change initiatives had similar results. Ninety- 
eight percent (98%) of Major Individualized TA 



Evaluation Highlights 2006-2009 



~ 3 ~ 



December 2009 



and 87% of Systems Change Initiatives resulted in 
changes in local systems, as reported by key 
contact(s) in the state. The following are examples 
of client feedback related to local systems changes: 

“We were shown how to gather our data ahead 
of time; how to develop protocols; how to ' drill 
down ' for the root cause or systemic causes of non- 
compliance and how to work with the provider in 
setting up procedures. ” 

“Local lead agency contracts continue to be 
evaluated and amended to better reflect their 
responsibility to provide general supervision and 
monitoring. " 

“[We have] stronger connections between early 
childhood special education and community based 
early childhood providers such as child care and 
Head Start through cross-sector trainings and 
conferences. " 

Practice Level Changes 

Ninety percent (90%) of external evaluation 
respondents agreed or strongly agreed that 
practices at the local level had been improved 
because of the services and/or products received 
from NECTAC. States reported changes in the 
areas of: 

• Data collection and monitoring participation (60%); 

• Direct service or teaching practices (51%); 

• Screening and/or assessment practices (37%); and 

• IFSP/IEP development (33%) 

The follow up data from Major Individualized TA 
and Long-Term Systems Change initiatives yielded 
somewhat broader results because three years of 
data were analyzed. Seventy-seven percent (77%) 
of Major Individualized TA and 100% of Systems 
Change Initiatives resulted in changes in practices 
that had impacts for children and families, as 
reported by key contact(s) in the state. 



Reported Impacts of NECTAC TA 


TYPE OF 
IMPACTS 


% of State 
Programs 
(External 
Evaluation) 


%of 

Major TA 


%of 

Systems 

Change 

Initiatives 


State 

System 


91% 


95% 


97% 


Local 

System 


91% 


98% 


87% 


Practice 

Changes 


90% 


77% 


100% 



Examples of client feedback related to practice 
level changes: 

“We have seen improvements in the capacity 
of staff in centers to develop and sustain quality 
inclusive programming and we even developed a 
Mentoring Program for Directors in order that they 
could develop more leadership in inclusion skills. ” 

“We definitely are using more family input as to 
functional skills, generalization of skills, etc. I 
think previously we focused primarily on scores 
and not so much on the functionality. " 

“[I] used Cara's Kit with providers to help them 
problem solve challenges in the classroom. One 
center is now building more on child interests to 
assist with helping children become more engaged 
with activities. " 

“Family reported data showed that there was an 
increase in the family's ability to effectively 
communicate their child's needs to others and their 
ability to help their child develop and learn from 
the previous period to the last report period. " 

CONCLUSIONS 



State Part C and Section 619 programs seek and 
receive TA from NECTAC on a regular basis in 
order to gain support in their implementation of 
IDEA. Overall, NECTAC TA is reported as high 
quality and relevant, and participants are able to 
use what they gain from the TA to make 
improvements to their systems for serving young 
children with disabilities and their families. 
Systems change initiatives have had the greatest 
reported impacts, as they are comprehensive, 
ongoing efforts designed to address state and local 
infrastructure, personnel development, and 
provider practices. 

Thank you and keep the feedback coming! 

NECTAC appreciates all the time and thoughtful 
answers our clients contributed to our evaluation. 
Client feedback is used regularly to improve our 
knowledge and TA services. For more information 
about the work of The National Early Childhood 
Technical Assistance Center, please visit our 
website at http://www.nectac.org 

The National Early Childhood Technical Assistance Center (NECTAC) 
is supported by cooperative agreement H326H060005 with the Office of 
Special Education Programs (OSEP), U.S. Department of Education 
(ED). NECTAC is a part of OSEP's Technical Assistance and 
Dissemination Network. This report is online at: 
http://www.nectac.org/~pdfs/pubs/nectac_eval_highlights_report.pdf 
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